An Integrated HCI Framework for Interpreting Meaningful Expressions

Integration of different modalities aims to increase the robustness of system and improves its performance in an unconstrained environment. With this motivation, the proposed research is focused on integrating two modalities such as hand gesture and posture in a unified framework where the criterion is measured by employing Particle Filter system. The proposed framework has two main modules: 1) gesture and posture recognition system, and 2) Particle filter based integration of these systems by incorporating Context Free Grammar rules to generate interpretation and inferences. In the recognition part, gesture classification is performed by HMM using hand orientation as a feature. However, the posture signs are firstly categorized into groups by fingertip detection for ASL finger spelling and then classified by SVM with statistical and geometrical features. The integration is performed by mapping the classification outcome on Particle Filter system approximating the probability density function that acts as contribution-weight at decision level. As the computed contribution-weights are independent of classification outcome, so it allows us to resolve classification ambiguities. Based on the contribution-weights, we have obtained a sequence of gesture and posture signs, which are used to infer the “meaningful expressions” by employing CFG for our test scenario. Experiments are conducted on 500 different combinations of restaurant orders with an overall 98.3% inference accuracy whereas classification accuracy of gesture and posture recognition is approximately 98.6% which proves the significance of proposed approach.